Discovering Objects in Dynamically-Generated Web Pages

As the web grows, more and more content is being hidden from the reach of traditional search engines. In this paper, we present THOR, a scalable and efficient tool to mine objects from this hidden web. With precision and recall over 90%, THOR automatically extracts objects of interest from dynamically-generated web pages. Then customized objectidentification algorithms are applied to locate the “interesting” objects in each page. We show that dynamicallygenerated pages tend to be a homogenous subset of pages found on the Web, and that these pages may be separated into distinct clusters of structurally-similar pages. Using this homogeneity across clusters along with traditional information retrieval techniques, we propose a two-phase clustering scheme consisting of a page clustering algorithm and a fragment clustering algorithm. Using this scheme, we can identify object-rich fragments of each page with an average of over 90% precision and over 95% recall.